Papers with DPO model
Model Extrapolation Expedites Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve LLM alignment training require expensive computational resources. |
| Approach: | They propose a model extrapolation method to expedite LLMs’ alignment with human preferences by amplifying parameter changes based on a first-order approximation without any additional training overhead. |
| Outcome: | The proposed method outperforms a fully-trained model on leading benchmarks and significantly outperformed open-source models. |
Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) often perform poorly in generating informative questions, as measured by expected information gain (EIG). |
| Approach: | They propose to use a large language model to enhance the informativeness of LLM-generated questions in 20-question game dialogues by applying a Direct Preference Optimization algorithm to generate low-EIG and high-EI questions. |
| Outcome: | The proposed method produces more effective questions even in domains different from those used to train the DPO model. |
Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | Best-of-N (BoN) sampling is an effective strategy for aligning Large Language Models (LLMs) to human preferences at the time of decoding. |
| Approach: | They propose a variant of BoN that incorporates the Minimum Bayes Risk objective as a proximity regularizer for BoN sampling. |
| Outcome: | The proposed method outperforms both BoN sampling and MBR decoding on the AlpacaFarm and Anthropic datasets. |
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for LLM alignment optimize tokens using a sparse, response-level reward or preference annotation. |
| Approach: | They propose an RLHF-equivalent distillation method for token-level reward optimization that incorporates the reward learned by DPO into the RLHG objective and builds a token-based teacher distribution. |
| Outcome: | The proposed method bridges the accuracy gap between the reward from the DPO model and the pure reward model by building a contrastive DPO reward with a normal and a reverse DPO. |